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FOREWORD 


An  important  part  of  the  research  of  the  Career  Development  and  Sol- 
dier Productivity  Technical  Area  of  the  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  (ARI)  supports  effective  career  management 
for  Army  officer  personnel.  As  part  of  this  concern  for  effective  career 
management,  ARI  is  conducting  research  on  a computer-based  career  infor- 
mation and  planning  system  for  Army  officers.  This  report  reviews  the 
types  of  outcome  measures  previously  used  in  career  counseling  research. 
The  immediate  purpose  of  the  review  is  to  provide  background  for  the  con- 
struction of  instruments  to  evaluate  a computer-based  career  information 
and  planning  system  for  Army  officers.  Other  related  reports  include  ARI 
Research  Memorandum  77-13,  which  provides  an  overview  of  career  develop- 
ment theory,  and  Research  Memorandum  77-14,  which  describes  a field  try- 
out of  the  computer-based  career  information  and  planning  system.  Contin- 
uing investigations  center  on  cost-benefit  analyses  of  the  system.  This 
research  is  conducted  under  Army  Project  2Q762717A766,  Manpower  Systems 
Management  (FY  78) » Task  C,  Career  Progression  Systems,  in  basic  support 
of  the  Office  of  Deputy  Chief  of  Staff  for  Personnel.  Dr.  John  0.  Crites 
and  Dr.  Clara  Hill  provided  comments  on  this  paper. 
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OUTCOME  MEASURES  FOR  CAREER  COUNSELING  RESEARCH 

r 

I 

BRIEF  1 


Requirements 

TO  provide  background  for  constructing  outcome  measures  (that  is, 
measures  of  the  effect  of  counseling)  to  evaluate  a career  information 
and  planning  system  for  Army  officers • 

Procedure : 

Previously  used  career  counseling  outcome  measures  (1950-76)  were 
classified  and  reviewed,  and  issues  emerging  from  the  review  were  dis- 
cussed. A set  of  recommendations  for  future  career  counseling  outcome 
research  addresses  these  issues. 


Findings: 

Career  counseling  researchers  have  demonstrated  a tendency  to  "re- 
invent the  wheel"  instead  of  building  on  each  other's  work.  A critique 
of  previous  career  counseling  research  revealed  the  following:  (1)  use 
of  inappropriate  criteria;  (2)  preponderance  of  self-report  measures,  as 
contrasted  to  the  use  of  measurement  approaches  such  as  cost  benefit 
analysis  or  behavioral  observation;  (3)  relative  infrequency  with  which 
reliability  and  validity  data  are  reported  for  the  instruments  used;  and 
(4)  methodological  shortcomings  concerning  randomization  of  subjects,  se- 
lection of  appropriate  comparison  groups,  and  choice  of  type  of  statisti- 
cal analysis.  Although  much  of  the  career  counseling  outcome  research 
has  been  characterized  by  one  or  more  of  these  inadequacies,  there  are 
many  examples  of  good  outcome  measurement  in  the  literature.  In  addition, 
most  published  research  demonstrates  some  useful  procedure  or  conceptual- 
ization which  would  be  helpful  to  career  counseling  researchers. 

Utilization  of  Findings: 

This  report  can  be  used  by  career  counseling  researchers  as  a re- 
source in  selecting  outcome  measures.  The  specific  application  is  for 
constructing  instruments  for  evaluating  a computer-based  career  informa- 
tion and  planning  system  for  Army  officers.  The  report,  however,  is  ex- 
pected to  have  broader  application  to  other  types  of  career  counseling 
approaches,  and  the  findings  presented  here  are  not  intended  to  be  lim- 
ited to  the  Army  officer  system. 
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OUTCOME  MEASURES  FOR  CAREER  COUNSELING  RESEARCH 


INTRODUCTION  * 

Placing  people  in  the  right  jobs  has  been  of  increasing  concern  in 
both  the  civilian  and  military  worlds.  As  stated  in  DA  Pamphlet  600-3, 

Officer  Professional  Development  and  Utilization,  objectives  of  the  Offi- 
cer Personnel  Management  Systems  are  to: 

1.  Develop  officers  in  the  right  numbers  and  with  the  right  skills 
to  satisfy  Army  requirements,  taking  maximum  advantage  of  inher- 
ent abilities,  aptitudes,  and  interests  of  the  individual  officer; 

2.  Assign  officers  according  to  the  Army's  needs  and  the  individual's 
competence  and  desires;  and 

3.  Improve  the  motivation  and  professional  satisfaction  of  the  offi- 
cer corps. 

One  way  of  implement ing  these  objectives  is  to  study  the  officer  ca- 
reer progression  system  and  design  interventions  that  enhance  the  career 
development  of  Army  officers.  Such  research  has  been  undertaken  by  the 
Career  Progression  Systems  Work  Unit  of  the  Career  Development  and  Soldier 
Productivity  Technical  Area  at  the  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  (ARI).  Now  underway  is  development  of  a 
computer-aided  career  planning  program  to  teach  officers  career  decision- 
making skills  and  also  provide  relevant  career  information. 

In  evaluating  any  career  development  program  such  as  described  above, 
decisions  must  be  made  concerning  which  of  a multitude  of  dependent  vari- 
ables are  to  be  used.  The  decision  must  be  made  on  which  variables  are 
best  suited  for  evaluating  that  particular  program  and  population.  There- 
fore, the  purpose  of  this  paper  is  to  provide  a review  of  the  types  of 
measures  previously  used  to  evaluate  career  counseling  outcome  research 
in  order  to  provide  background  for  a specific  application:  constructing 
instruments  for  evaluating  a computer-based  career  information  and  plan- 
ning system  for  Army  officers.  However,  these  issues  are  considered  in 
the  broader  context  of  the  working  world,  and  the  findings  presented  here 
are  not  intended  to  be  limited  to  the  Army  officer  system. 

The  discussion  begins  with  some  general  considerations  bearing  on  se- 
lecting dependent  variables  for  outcome  research,  and  a summary  of  the 
most  frequently  used  measures  follows  this  discussion.  Next,  some  issues 
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Vhe  author  wishes  to  express  her  appreciation  to  Dr.  John  O.  Crites  and 
to  Dr.  Clara  Hill  for  their  helpful  comments  on  this  paper. 
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that  have  emerged  from  the  career  counseling  outcome  research  are  de- 
scribed.  Finally,  recommendations  for  resolving  some  of  the  problems  of 
outcome  measurement  are  considered. 

GENERAL  CONSIDERATIONS 

Several  general  factors  need  to  be  considered  in  deciding  which  of 
many  dependent  variables  are  to  be  selected  for  career  counseling  outcome 
research.  One  such  factor  involves  the  type  of  behavior  to  be  assessed. 
What  are  the  behaviors?  Are  they  short-range  or  long-range  in  nature? 

Must  we  rely  on  self-report,  or  are  unobtrusive  measures  practical?  In 
addition,  where  are  these  behaviors  to  be  observed?  Will  this  observa- 
tion be  conducted  in  the  field,  in  the  laboratory,  or  through  a mail  sur- 
vey? The  location  may  impose  limits  in  terms  of  the  available  sample, 
whether  randomization  is  possible,  etc.  Where  the  research  will  take 
place  may  also  influence  consideration  of  when  it  can  occur.  Timing  may 
control  the  number  and  type  of  researchers  and  instruments  to  be  used,  as 
well  as  the  possibility  of  followup  studies.  How  these  behaviors  are  to 
be  observed  is  also  an  important  consideration,  which  relates  to  decisions 
about  the  behaviors  selected  and  where  and  when  the  observations  are  to 
be  made.  What  is  the  availability  of  current  instruments  (human  observers 
as  well  as  paper-and-pencil  instruments)?  Do  instruments  of  proven  relia- 
bility and  validity  exist,  or  must  they  be  constructed?  Is  the  coopera- 
tion of  potential  observers  or  sources  of  information  assured?  To  what 
extent  are  appropriate  records  obtainable? 

Some  of  the  specific  factors  involved  in  outcome  measures  have  been 
mentioned  in  general  terms  of  what,  where,  when,  and  how.  It  is  the  "why," 
however,  that  often  determines  what,  where,  when,  and  how  the  research  is 
to  be  carried  out.  Objectives  of  a field  test,  for  example,  may  differ 
from  those  of  program  evaluation.  As  Helliwell  and  Jones  (1975)  have 
pointed  out,  field  tests  are  like  "dress  rehearsals”;  that  is,  the  main 
purpose  of  a field  tryout  is  to  "debug"  the  system.  From  their  experience 
with  evaluating  a model  for  inproving  guidance  programs,  Helliwell  and 
Jones  (1975)  made  the  following  suggestions  for  field  tests: 

1.  To  investigate  interim  rather  than  terminal  behaviors, 

2.  To  use  fewer  and  shorter  evaluation  instruments, 

3.  To  keep  very  careful  records  (including  anecdotal  ones)  concern- 
ing the  difficulties  encountered  in  conducting  the  test,  and 

4.  To  obtain  strong  assurance  of  cooperation  and  real  involvement 
from  people  at  the  test  site  location. 

However,  if  the  purpose  of  the  test  is  not  merely  to  debug  a system 
but  to  evaluate  the  efficacy  of  the  system  or  to  compare  it  with  another 
system  or  systems,  the  approach  may  be  somewhat  different.  In  a critique 
of  process  and  outcome  research  in  counseling,  Kelley,  Smits,  Leventhal, 
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and  Rhodes  (1970)  concluded  that  the  two  major  shortcomings  of  the  73  stud- 
ies they  reviewed  were  the  failure  to  include  control  groups  in  the  exper- 
imental design  and  the  lack  of  randomization*  In  terms  of  the  Campbell 
and  Stanley  (1966)  classification  system,  most  of  the  research  reviewed  by 
Kelley  et  al.  involved  one-shot  case  studies,  one  group  pretest-posttest, 
or  static-group  conparisons — none  of  which  constitutes  a true  experimental 
design. 


Hilton's  (1974)  criteria  for  adequate  evaluation  of  career  education 
and  guidance  outcomes  also  emphasized  random  assignment  to  treatments. 

In  addition,  Hilton  suggested  pretest-posttest  measurement,  a representa- 
tive sample,  a sample  large  enough  to  obtain  statistically  reliable  re- 
sults, and  an  "objectively  scored  measure  of  relevant  characteristics  of 
known  reliability  and  validity"  (Hilton,  1974).  The  issue  of  reliability 
and  validity  of  the  instruments  is  an  important  one  in  career  counseling 
outcome  research,  and  the  topic  will  be  discussed  in  some  detail  later  in 
this  paper. 


In  this  discussion  of  factors  to  be  considered  in  selecting  career 
counseling  outcome  measures,  it  has  been  suggested  that  the  purpose  of 
the  research  is  of  primary  importance.  The  objectives  to  be  accomplished 
will  help  the  researcher  determine  which  dependent  variables  are  to  be 
used.  The  references  mentioned  above  are  good  background  sources  for  in- 
vestigators faced  with  the  problem  of  selecting  career  counseling  outcome 
variables. 


In  addition,  Bingham's  (1974)  chapter  on  assessing  counseling  outcome 
may  be  helpful.  Although  the  Bingham  paper  is  primarily  directed  toward 
counselors  concerned  with  career  guidance  and  career  education  in  schools, 
portions  of  his  discussion  are  pertinent  to  any  developmental  career  plan- 
ning program.  Another  valuable  (and  more  technical)  source  for  the  re- 
searcher interested  in  evaluation  is  the  two-volume  Handbook  of  Evaluation 
Research.  The  first  volume  (Struening  & Guttentag,  1975)  emphasizes  strat- 


egies and  methods  of  evaluation,  whereas  the  second  volume  (Guttentag  & 
Struening,  1975)  focuses  on  evaluation  in  specific  content  areas. 


SUMMARY  OF  PREVIOUSLY  USED  MEASURES 


In  a review  of  outcome  research  in  educational  and  vocational  coun- 
seling, Myers  (1971)  has  classified  dependent  variables  for  educational- 
vocational  decisionmaking  and  effective  role  functioning.  For  educational- 
vocational  decisionmaking,  dependent  measures  fell  into  three  categories: 
accuracy  of  self-knowledge,  appropriateness  of  career  preference  or  choice, 
and  instrumental  behaviors.  Variables  related  to  effective  role  function- 
ing were  classified  according  to  whether  they  were  criteria  of  adjustment 
or  performance. 


Adapting  the  Myers  classification  system  somewhat,  this  report  discus- 
ses dependent  variables  for  career  counseling  outcome  research  in  terms  of 
the  following  categories: 


: 
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1.  Career  decisionmaking  measures 

a.  Accuracy  of  self-knowledge 

b.  Appropriateness  of  choice 

c.  Instrumental  behaviors  (career  information  seeking,  securing 
job) 

d.  Attitudes  toward  choice  (certainty,  satisfaction,  commitment, 
career  salience) 

2.  Effective  role  functioning  measures 

a.  Performance  variables  (academic  performance,  career-related 
knowledge) 

b.  Adjustment  variables  (career  maturity,  self-concept  changes) 

3.  Ratings  of  counseling  satisfaction  and  effectiveness 

4.  Miscellaneous  measures 

To  clarify  the  discussion  that  follows,  a table  is  presented  for 
each  type  of  dependent  measure  enumerated  above.  For  each  study  using 
that  particular  kind  of  measure,  the  table  identifies  the  criterion  or 
specific  variable  involved  and  describes  briefly  how  the  assessment  was 
made.  (The  studies  in  Tables  1-8  cover  the  period  1950-76.) 

Note  that  this  paper  is  limited  to  career  counseling  outcome  re- 
search. Research  concerned  only  with  educational  counseling  is  not  in- 
cluded. Also,  the  focus  is  on  career  counseling  outcome  variables,  even 
though  process  and  outcome  cannot  always  be  easily  differentiated.  In 
fact,  Grummon  (1972)  has  argued  that  process  studies  involving  a series 
of  measurements  during  the  course  of  counseling  are  also  outcome  studies. 
Grummon  would  consider  the  trend  line  for  an  increase  in  self-acceptance, 
for  example,  an  outcome  measure.  However,  process  variables  such  as 
client-counselor  rapport,  or  the  specific  topics  covered  during  a series 
of  career  counseling  interviews,  are  not  dealt  with  here.  The  discussion 
focuses  on  variables  that  assess  the  inpact  of  career  counseling  after 
counseling  has  been  completed  (although,  of  course,  pretest  measures  may 
be  made  in  order  to  determine  the  change  that  has  occurred  during  the 
period  of  counseling) . 

Also  not  included  are  studies  that  relate  to  career  counseling  out- 
come variables  but  that  do  not  actually  involve  counseling.  One  such 
type  of  research  comprises  investigations  of  test  score  recall.  Although 
these  studies  have  typically  used  outcome  measures,  they  are  not  included 
unless  the  test  interpretation  was  part  of  a career  counseling  treatment. 
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Lastly,  the  summary  of  dependent  variables  used  in  career  counseling 
outcome  research  is  not  intended  to  be  an  exhaustive  compendium.  Rather, 
the  purpose  of  the  summary  is  to  describe  the  kinds  of  measures  typically 
used  in  career  counseling  outcome  research  and  to  present  some  examples 
of  each  type. 


Career  Decisionmaking  Measures 

Career  decisionmaking  measures  constitute  the  first  major  category 
of  career  counseling  outcome  variables.  Pour  types  of  dependent  measures 
are  included:  (a)  accuracy  of  self-knowledge,  (b)  appropriateness  of 
choice,  (c)  instrumental  behaviors,  and  (d)  attitudes  toward  career 
choice. 

Accuracy  of  Self-Knowledge.  Variables  of  this  type  assess  the  dis- 
crepancy between  client  self-estimates  of  a characteristic  and  a criter- 
ion, with  the  criterion  consisting  of  a test  score  or  a rating  by  judges. 
The  usual  self-estimate  procedure  involves  (a)  having  subjects  estimate 
their  standing  on  the  dimension  of  interest  before  any  treatment  (coun- 
seling) takes  place,  (b)  obtaining  similar  estimates  after  counseling, 
and  (c)  determining  the  effect  of  the  treatment  by  analyzing  the  pretest- 
posttest  discrepancy  (between  self -estimates  and  criterion)  and/or  dif- 
ferences between  experimental  and  control  groups.  Table  1 summarizes  the 
studies  that  have  included  self-knowledge  as  a dependent  measure. 

Many  career  counseling  outcome  studies  have  used  self-knowledge  var- 
iables. Typical  of  such  outcome  research  is  an  early  study  by  Johnson 
(1953).  In  this  investigation,  male  subjects  were  administered  tests  of 
scholastic  aptitude  (Otis),  interests  (Kuder),  and  personality  (Bernreuter). 
Prior  to  seeing  a counselor  for  an  initial  interview,  the  subject  re- 
sponded to  a questionnaire  on  which  he  estimated  in  which  fifth  of  the 
population  he  stood  with  respect  to  intelligence,  interests,  and  person- 
ality. Immediately  after  treatment  (vocational  counseling),  the  subject 
again  completed  the  questionnaire  containing  the  self -estimates.  In  a 
1-month  followup,  the  subjects  filled  out  the  questionnaire  for  a third 
time.  At  the  conclusion  of  the  counseling,  the  subject's  "true"  standing 
on  the  various  dependent  variables  was  rated  by  his  counselor,  who  used 
the  test  results  in  determining  the  rating. 

As  can  be  seen  in  Table  1,  outcome  studies  using  self-knowledge  mea- 
sures as  dependant  variables  have  tended  to  focus  on  self-estimates  of  in- 
terests and  aptitudes.  Measures  of  achievement  and  personality  have  been 
used  less  frequently.  Most  of  the  studies  ask  the  subjects  to  compare 
others  with  themselves  and  estimate  into  which  proportion  (usually  quin- 
tile) of  the  comparison  population  they  would  fall.  A slightly  different 
procedure  was  used  by  Guttman  and  Haase  (1972),  who  had  subjects  estimate 
their  three  highest  and  three  lowest  (Kuder)  interest  areas. 
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With  few  exceptions,  the  studies  given  in  Table  1 used  test  scores 
as  criteria.  Three  studies  used  judges ' ratings  as  criteria.  Even  when 
judges'  ratings  were  used,  however,  judges  had  had  test  data  available 
to  them.  Johnson  (1953),  for  example,  explained  that  his  counselors' 
i judgments  were  "guided  but  not  determined  solely  by  the  test  results" 

(p.  333).  Overwhelmingly,  then,  career  counseling  outcome  research  has 
depended  on  test  scores,  directly  or  indirectly,  as  criteria  for  client 
self -estimates . 

In  the  Kamm  and  Wrenn  (1950)  study,  a variant  of  the  self-knowledge 
criterion  was  used.  Using  summaries  of  pre-interview  and  post-interview 
data,  three  judges  rated  client  "acceptance"  of  test  information. 

Appropriateness  of  Career  Choice.  "Appropriateness"  of  career 
choice,  sometimes  termed  "realism"  or  "wisdom,"  is  another  commonly  used 
measure  of  career  counseling  outcome.  Such  variables  are  based  on  assump- 
tions that  individuals  are  better  suited  for  some  careers  than  others  to 
a measurable  degree  of  "appropriateness."  The  usual  procedure  is:  (a)  to 
assess  the  client's  characteristics  that  are  vocationally  relevant,  (b)  to 
determine  the  requirements  of  the  chosen  career  in  terms  of  these  charac- 
teristics, and  (c)  to  estimate  the  degree  of  congruence  between  client 
characteristics  and  career  requirements.  Table  2 summarizes  some  of  the 
research  that  has  dealt  with  appropriateness  as  a dependent  variable. 

The  most  common  method  for  determining  whether  a career  is  appropri- 
ate for  an  individual  has  been  judges'  ratings.  Judges  may  make  their 
ratings  on  the  basis  of  case  materials  and  have  no  contact  with  the  sub- 
jects. Or  the  judges  may  be  the  counselors  actually  administering  the 
career  counseling  and,  thus,  use  interview  data  as  well  as  the  usual  test 
I information  and  questionnaire  responses.  Judges'  ratings  have  been  of 

various  kinds.  Some  authors  have  used  dichotomous  ratings,  such  as  real- 
istic/unrealistic (Hewer,  1959;  Hoyt,  1955;  Wright,  1963).  Other  research 
has  used  scales  that  represented  a degree  of  appropriateness  from  realis- 
tic to  unrealistic  (Apostal,  1960;  Bivlofsky  et  al. , 1953;  Hewer,  1966; 
Mencke  & Cochran,  1974;  Westbrook,  1967).  A few  researchers  (Hanson  & 
Sander,  1973;  Pilato  & Myers,  1975)  have  constructed  instruments  ranging 
from  one  extreme  of  inappropriateness  to  another,  with  the  midpoint  repre- 
senting an  optimal  degree  of  realism. 

* A typical  study  using  appropriateness  of  choice  as  a dependent  vari- 

able is  that  of  Hanson  and  Sander  (1973).  In  this  research,  three  exper- 
ienced counselors  judged  the  subjects'  "realism  of  vocational  plans"  on  a 
5-point  scale  that  included  the  following  categories:  overshooting,  mod- 
erately overshooting,  realistic,  moderately  undershooting,  and  undershoot- 
ing. Overshooting  was  defined  as  aspiring  to  an  occupation  that  was  be- 
yond the  person's  ability  level,  and  undershooting  involved  the  judgment 
that  the  subject  could  probably  succeed  in  a higher  level  occupation.  Each 
judge  was  provided  with  aptitude,  interest,  and  achievement  test  scores 
for  the  subjects  as  well  as  information  on  their  grades,  extracurricular 
activities,  work  experience,  health,  and  family  background.  Hanson  and 
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Sander  (1973)  also  reported  Interjudge  reliabilities,  a practice  not  al- 
ways adhered  to  in  career  counseling  outcome  research. 

Although  judges'  ratings  are  the  most  frequently  used  means  of 
assessing  appropriateness  of  career  choice,  some  authors  have  developed 
techniques  that  are  objective  in  nature  and  avoid  the  problem  of  the  re- 
liability of  such  ratings.  Zener  and  Schnuelle  (1976)  compared  the  Holland 
summary  codes  of  occupations  their  subjects  were  considering  with  the  sub- 
jects' codes  obtained  from  the  SDS  or  the  VPI.  Mencke  and  Cochran  (1974) 
used  the  Holland  codes  for  the  career  choice  but  conpared  them  with  the 
codes  for  the  subjects'  measured  interests.  Still  another  measure  was 
used  by  Westbrook  (1967),  who  conpared  aptitude  scores  with  the  aspiration 
level  of  the  choice.  Pilato  and  Myers  (1975)  quantified  appropriateness 
in  terms  of  the  client's  measured  characteristics  (aptitude  and  interests) 
and  Roe's  (1957)  classification  of  occupations  by  level  and  field.  As  did 
Hanson  and  Sander  (1973)  in  the  study  described  above,  Pilato  and  Myers 
took  into  account  the  fact  that  an  aptitude  level  higher  than  that  re- 
quired for  a job  may  be  just  as  inappropriate  as  an  aptitude  level  that 
is  too  lew. 

All  but  one  of  the  studies  reported  in  Table  2 used  judges'  ratings 
or  test  scores  as  the  criterion.  The  single  exception  was  a study  by  Hewer 
(1966),  who  used  the  current  occupation  of  her  subjects  as  a criterion. 

Instrumental  Behaviors.  Almost  all  measures  falling  into  the  classifi- 
cation of  instrumental  behaviors  involve  career  or  educational  information- 
seeking behavior.  As  shown  in  Table  3,  the  most  commonly  used  dependent 
measures  are  frequency  and  variety  of  information  seeking.  The  frequency 
variable  involves  a count  of  the  total  number  of  contacts  made  by  the  cli- 
ent to  obtain  career  information,  and  variety  is  the  number  of  different 
types  of  sources  (libraries,  counselors,  etc.)  the  subject  consulted.  Self- 
report  procedures  are  typically  used  with  the  data  obtained  by  use  of  a 
questionnaire  or  a structured  interview.  An  unobtrusive  measure  of  infor- 
mation seeking  was  used  by  Cooper  (1976),  who  recorded  the  number  of  sub- 
jects who  returned  to  the  counseling  center  to  pick  up  career  information 
after  career  counseling  had  been  completed. 


Almost  all  of  the  measures  of  information-seeking  behavior  appeared 
to  be  identical  to  or  derived  from  those  developed  by  Krumboltz  and  his 
associates.  (Borman,  1972,  and  Samaan  and  Parker,  1973,  were  the  only  in- 
vestigators who  did  not  identify  their  questionnaires  as  being  related  to 
those  used  by  Krumboltz. ) Most  of  these  procedures  involved  a structured 
interview  such  as  that  described  in  Krumboltz  and  Schroeder  (1965).  An 
interviewer  who  did  not  know  which  treatment  the  person  had  received  in- 
terviewed each  subject  a few  weeks  after  counseling.  The  interviewer  fol- 
lowed a detailed  questionnaire  constructed  to  elicit  reports  of  21  cate- 
gories of  information-seeking  behavior,  such  as  reading  books  or  magazine 
articles  about  careers,  requesting  an  occupational  pamphlet,  talking  to 
persons  working  in  a particular  occupation,  watching  TV  shows  or  listening 
to  radio  programs  about  occupations,  etc.  Krumboltz  and  Schroeder  (1965) 
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of  reports  from  several  investigations  of  computer-aided  guidance  systems 
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the  subjects'  reports  by  checking  library  records,  TV  and  radio  program  1 | 

guides,  etc.  Most  investigators  using  structured  interviews  have  at- 
tempted to  verify  the  self-reports  of  random  samples  of  subjects,  and 

they  have  concluded  that  such  interviews  are  highly  valid  (Krumboltz  & , 

Thoresen,  1964;  Samaan  & Parker,  1973;  Thoresen  & Hamilton,  1972;  Thoresen  r 

& Krumboltz,  1967;  Thoresen  & Krumboltz,  1968;  Thoresen,  Hosford,  & 

Krumboltz,  1970). 

Zener  and  Schnuelle  (1976)  modified  the  Krumboltz  and  Schroeder  1 

(1965)  checklist  to  expand  the  range  of  information-seeking  behaviors  and 
administered  the  resulting  instrument  as  part  of  a followup  questionnaire. 

Cooper  (1976)  used  the  same  instrument,  and  Krivatsy  and  Magoon  (1976) 
used  an  adaptation  of  the  Zener  and  Schnuelle  questionnaire.  Jones  and 
Krumboltz  (1970)  also  used  a questionnaire  similar  to  the  Krumboltz  and 

Schroeder  (1965)  interview  schedule.  ' 

Only  one  of  the  studies  in  Table  3 (Thoresen,  Krumboltz,  & Varenhorst, 

1967)  assessed  an  instrumental  behavior  other  than  career  information  seek- 
ing. In  the  course  of  individual  interviews  held  3 weeks  after  a single  j 

career  counseling  session,  Thoresen  et  al.  ( 1967)  determined  whether  each 
subject  had  obtained  (or  made  definite  plans  to  obtain)  a summer  or  part- 
time  job  connected  with  an  occupation  being  considered. 

Attitudes  Toward  Choice.  Several  attitudes  toward  career  and  educa- 
tional preferences  and  goals  are  included  in  this  category.  As  shown  in 
Table  4,  the  most  commonly  assessed  variable  is  certainty  of  career  choice 
(sometimes  called  "decidedness"  or  "commitment").  Two  studies  (Hewer, 

1959;  Zener  & Schnuelle,  1976)  also  measured  satisfaction  with  the  career 
choice.  Barak,  Carney,  and  Archibald  (1975)  used  a rating  technique  that 

is  typical  of  the  certainty  and  satisfaction  measures.  Two  items  on  their  ' 

career  assessment  form  had  5-point  scales  ranging  from  a score  of  "1"  for 
"completely  undecided"  (about  a career  or  major)  to  a score  of  "5"  repre- 
senting "completely  decided."  Also  included  in  Table  4 is  Cooper's  (1976) 
career  salience  variable.  Career  salience,  which  is  the  importance  of  ca- 
reer (rather  than  the  importance  of  a specific  career)  to  the  subject, 
was  assessed  by  Angrist's  (1972)  Lifestyle  Index. 


Effective  Role  Functioning  Measures 

A second  major  category  of  career  counseling  outcome  variables  con- 
cerns what  Myers  (1971)  has  called  "effective  role  functioning."  These 
measures  are  further  divided  into  "performance"  variables  and  "adjustment" 
variables. 

Performance  Variables.  Dependent  variables  of  this  type  are  concerned 
with  academic  performance  or  the  assessment  of  some  type  of  knowledge. 

Most  counseling  research  using  academic  performance  as  a criterion  has  been 
educational  counseling,  and,  thus,  it  is  not  covered  in  this  paper.  In  his 
review,  Myers  (1971)  has  noted  that  grade  point  average  (GPA)  proved  to  be 
a disappointing  outcome  measure.  Hill  and  Grieneeka  (1966),  in  a comparison 
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of  students  seeking  (and  not  seeking)  educational  and  vocational  counsel- 
ing, found  no  differences  between  the  two  groups  in  GPA  or  rate  of  gradu- 
ation. 


Most  dependent  variables  of  the  performance  type  have  been  measures 
of  some  kind  of  career-related  knowledge.  As  shown  in  Table  5,  the  ca- 
reer knowledge  being  evaluated  may  involve  career  information  in  terms  of 
vocational  and  educational  opportunities  and/or  requirements,  knowledge 
of  tests  and  testing,  or  knowledge  of  career  development  theory.  Most 
researchers  used  a quiz  of  some  sort  to  measure  career  knowledge.  In 
addition  to  using  a quiz,  Thoresen  and  Hamilton  (1972)  prepared  simula- 
tion test  booklets  that  contained  short  descriptions  of  situations  involv- 
ing career  exploration.  After  reading  each  vignette,  subjects  were  asked 
to  specify  how  they  would  explore  and  evaluate  the  job  involved.  Both 
Krivatsy  and  Magoon  (1976)  and  Zener  and  Schnuelle  (1976)  asked  subjects 
to  recall  their  Holland  codes.  Other  measures  of  knowledge  of  Holland's 
theory,  not  clearly  described,  also  were  used  in  these  two  studies. 

Adjustment  Variables.  There  are  two  major  subdivisions  of  adjust- 
ment variables:  measures  of  career  maturity  and  measures  related  to 
self-concept.  Studies  using  the  two  types  of  variables  are  reported 
in  Table  6. 

Researchers  who  have  used  career  maturity  as  a dependent  variable 
have  tended  to  excerpt  items  from  instruments  designed  to  measure  this 
construct.  Crites'  Career  Maturity  Inventory  (formerly  the  Vocational 
Development  Inventory),  Super's  Career  Development  Inventory,  and 
Cooley  and  Lownes ' Readiness  for  Vocational  Planning  have  been  the 
instruments  most  frequently  used.  Some  investigators  have  used  the 
entire  instrument  (Myers,  Lindeman,  Thompson,  & Patrick,  1975;  Swails 
& Herr,  1976)  or  have  attempted  to  check  the  adequacy  of  the  adapted 
instrument  (Mencke  & Cochran,  1974;  Perrone  & Kyle,  1975).  Smith  and 
Evans  (1973)  used  the  Vocational  Decision  Checklist,  which  classified 
respondents  into  one  of  four  stages  of  vocational  development  (explo- 
ration, crystallization,  choice,  or  clarification). 

The  other  major  category  of  adjustment  variables  includes  mea- 
sures that  relate  to  the  client's  self-concept.  One  such  measure,  the 
"incorporation  score,"  has  been  developed  by  Healy  (1968).  The  incor- 
poration construct  is  based  on  the  premise  that  career  choice  is  an 
implementation  of  the  self-concept  (Super,  1957).  Healy  defines  incor- 
poration as  the  degree  of  similarity  between  a person's  self-rating  and 
his  or  her  rating  of  an  occupation  on  the  same  dimensions.  Specifically, 
the  subjects  use  7-point  semantic  differential  scales  to  rate  both  them- 
selves and  an  occupation  on  (a)  25  occupationally  relevant  traits  and 
(b)  22  bipolar  personality  traits.  The  "incorporation  score"  is  ob- 
tained by  summing  the  differences  between  the  self-ratings  and  the 
occupational  ratings  for  each  of  the  two  sets  of  traits,  disregarding 
the  direction  of  the  differences  and  subtracting  the  total  from  100 
(score  = 100  - Z |x  | ).  Thus,  higher  incorporation  scores  represent 
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greater  "incorporation"  (greater  similarity  between  self  and  occupa- 
tion) because  the  summed  differences  are  subtracted  from  100,  a proce- 
dure that  results  in  higher  scores  for  lesser  differences.  Using  the 
same  procedure,  Healy,  Bailey,  and  Anderson  (1973)  used  range  of 
incorporation  scores  as  a dependent  variable. 

Several  studies  by  Williams  and  his  colleagues  (Williams,  1962; 
Hills  & Williams,  1965;  Williams  & Hills,  1962)  have  used  two  self- 
concept  variables  based  on  the  Butler  and  Haigh  (1954)  Q-sort:  the 
"self-adjustment  score"  and  the  "congruence  score."  Subjects  were 
asked  to  sort  the  74  Butler  and  Haigh  items  into  two  piles — "less 
like  me"  and  "more  like  me."  Three  sorts  were  made.  The  first  sort 
was  according  to  the  subject's  current  view  of  himself  or  herself, 
the  second  sort  was  in  terms  of  the  "ideal"  person,  and  the  last  sort 
was  a description  of  the  "ordinary"  person  or  student.  The  self-ad- 
justment  score  was  the  sum  of  the  number  of  "good  adjustment"  items 
placed  in  the  "more  like  me"  category  and  the  number  of  "poor  adjust- 
ment" items  placed  in  the  "less  like  me"  direction.  The  congruence 
score  was  obtained  by  counting  the  number  of  items  sorted  identi- 
cally for  each  pair  of  concepts  (Self-Ideal,  Self-Ordinary , Ideal- 
Ordinary)  and  taking  the  mean  of  the  three  scores. 

In  Tipton's  (1969)  research,  subjects  were  asked  to  rate  eight 
concepts  relating  to  their  academic  ability  or  to  their  chances  of 
achieving  personal  satisfaction  in  a given  academic  area.  The 
semantic  differential  technique  was  used,  and  analyses  were  con- 
ducted to  compare  the  pretest-posttest  differences  of  the  concepts 
for  various  groups  of  subjects.  Another  self-concept  measure  used 
to  assess  counseling  outcome  was  an  adaptation  of  Holland  and  Baird's 
(1968)  Interpersonal  Competency  Scale.  Zener  and  Schnuelle  (1976) 
used  a shortened  version  of  this  instrument,  which  asks  the  respon- 
dent for  information  or  self-ratings  concerning  traits,  life  goals, 
home  background,  interests,  competencies,  and  high  school 
achievements. 


Ratings  of  Counseling  Satisfaction  and  Effectiveness 

The  counseling  outcome  measures  described  in  this  section  assess 
counseling  satisfaction  and  the  perception  of  counseling  effectiveness. 

As  will  be  apparent  from  the  descriptions  of  some  of  the  measures,  re- 
searchers have  not  always  clearly  differentiated  between  the  twc  vari- 
ables. Measures  of  perceived  effectiveness,  for  example,  have  been 
interpreted  as  indicators  of  satisfaction.  Also,  a number  of  different 
terms  have  been  used  for  satisfaction  or  effectiveness.  The  "enthusiasm" 
variable  of  Barahal,  Brammer,  and  Shostrom  (1950)  can  be  considered  a 
satisfaction  measure,  whereas  usefulness  or  helpfulness  of  information, 
e.g. , Babcock  and  Kaufman  (1976)  and  Holmes  (1964),  pertains  to  effec- 
tiveness. Table  7,  which  summarizes  outcome  research  using  ratings  of 
satisfaction  or  effectiveness,  shows  that  almost  all  the  studies  have 
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used  client  self-ratings.  Only  a few  studies  (Barahal  et  al.,  1950; 

Forgy  & Black,  1954;  Healy,  1973)  used  judges'  ratings. 

Ratings  of  satisfaction  and  effectiveness  may  involve  a global,  , 

overall  measure  of  the  variable  of  interest,  or  the  ratings  may  assess  t 

specific  facets  of  the  variable.  A global  measure  may  consist  of  a 
single  item,  or  it  may  represent  an  average  or  summation  of  two  or  more 
items.  Barahal  et  al.  (1950)  used  a global  measure  of  satisfaction  de- 
fined as  an  "overall  f e®ling-tone  rating."  This  rating  was  made  by  1 

three  judges  who  listened  to  recordings  of  structured  interviews  in 
which  the  interviewer  covered  20  items  that  dealt  with  such  aspects  as 
clarity  of  future  vocational  plans  and  helpfulness  of  conferences  with 
the  counselor.  Although  the  judges  rated  each  item  separately  on  a 
5-point  scale  (from  "very  enthusiastic"  to  "dissatisfied  and/or  highly 
critical"),  they  also  estimated  an  "overall  feeling-tone  rating"  using 
the  same  5-point  scale.  An  example  of  a global  measure  resulting  from 
the  summation  of  items  is  Zener  and  Schnuelle's  (1976)  "Evaluation 
Scale."  These  researchers  combined  two  highly  correlated  (r  = .82) 
items  ( "I  would  recommend  the  SDS  to  a friend"  and  "My  summary  code 
seems  reasonable")  to  obtain  their  global  measure  of  satisfaction/ 
effectiveness. 

! 

Ratings  may  also  involve  specific  rather  than  global  measures  of 
satisfaction  or  effectiveness.  Examples  of  specific  outcomes  used  to 
assess  satisfaction  or  effectiveness  are  "to  recommend  the  vocational 
exploration  experience  to  a friend"  (Cooper,  1976),  "setting  up  goals 
consistent  with  one's  abilities  and  interests"  (Graff,  Danish,  & 

Austin,  1972),  and  "clarification  of  career  direction"  (Healy,  1973). 

Linden,  Stone,  and  Shertzer  (1965)  have  devised  an  instrument 
that,  although  not  specifically  designed  for  career  counseling,  could 
be  used  as  an  outcome  measure  of  satisfaction  with  career  counseling. 

The  Counseling  Evaluation  Inventory  developed  by  Linden  et  al.  contains 
a subset  of  items  that  assesses  client  satisfaction.  Some  satisfaction 
items  relate  to  process  variables,  but  four  items  are  concerned  with 
outcome  and  are  enumerated  in  Table  7.  In  addition,  the  authors  have 
reported  final  scoring  weights,  factor  loadings,  test-retest  reliabil- 
ity coefficients,  social  desirability  ratings,  and  validity  data  for 
the  instrument. 

Also,  it  might  be  noted  that  although  satisfaction  and  perceived 
effectiveness  are  considered  outcome  measures,  some  researchers  base 
their  assessment  on  process  variables.  Wright  (1963),  for  example, 
obtained  client  ratings  of  the  following:  warmth  of  the  relationship, 
coverage  of  the  test  information,  clarity  of  the  test  interpretation, 
and  value  of  the  test  interpretation  for  educational  and  vocational 
planning.  Of  these  items,  only  the  last  is  clearly  an  outcome  measure. 

The  assumption  is  that  the  warmer  the  counselor-client  relationship, 
the  more  complete  the  coverage  of  the  test  information,  etc. , then  the 
more  effective  the  counseling. 
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Miscellaneous  Measures 

There  are  several  miscellaneous  variables  that  do  not  fit  into  the 
categories  previously  described.  These  dependent  variables  are  summa- 
rized in  Table  8. 

One  measure  seldom  used  in  career  counseling  research  is  the  cost 
analysis  conducted  by  Krivatsy  and  Magoon  (1976).  These  authors  calcu- 
lated the  mean  cost  per  subject  in  terms  of  personnel  time,  personnel 
cost,  materials  cost,  and  a total  cost  (personnel  cost  plus  materials 
cost).  Barahal  et  al.  (1950)  also  reported  costs  for  (a)  "the  mean 
dollar  estimate  of  the  value  of  the  guidance  service"  and  (b)  the 
actual  costs  for  tests  and  personnel  time. 

Another  infrequently  used  variable  is  Anderson  and  Binnie's  (1971) 
measure  of  occupational  aspiration,  which  these  researchers  adapted  from 
Haller  and  Miller's  (1963)  Occupational  Aspiration  Scale.  On  this  scale, 
the  stuaent  is  asked  to  select  from  jobs  at  10  different  occupational 
levels.  Selections  are  made  for  realistic  and  idealistic  preferences,  as 
well  as  for  short-range  and  long-range  choices.  The  prestige  of  each  of 
the  subject's  selections  then  is  assessed  following  the  Haller  and  Miller 
procedure. 

A more  commonly  used  outcome  measure  is  number  of  career  choice 
options.  For  this  criterion,  the  subject  is  asked  to  s’tate  the  career 
alternatives  he  or  she  is  currently  considering,  and  a count  is  made  of 
the  number  of  options.  Before-  and  after-measures  (Mencke  & Cochran, 
1974)  of  the  number  of  options  also  may  be  used.  There  are  some  prob- 
lems with  using  this  kind  of  variable  as  an  outcome  measure,  because  a 
larger  number  of  options  may  or  may  not  be  desirable  for  a particular 
client.  A somewhat  similar  measure  is  to  ask  the  subject  to  report  the 
amount  of  time  spent  thinking  about  occupations  or  about  self  during  a 
given  period  of  time  (Krivatsy  and  Magoon,  1976;  Zener  and  Schnuelle, 
1976).  Again,  the  assumption  is  that  "more  is  better" — that  is,  the 
more  time  an  individual  thinks  about  career-related  concerns,  the 
better  that  person's  career  decisions  will  be. 
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Foreign  & James  (1973)  Change  in  test  scale  Pre-post  measures  of  relevance  of  interest,  and  person- 

relevance  ality  test  scales 


ISSUES  IN  CAREER  COUNSELING  OUTCOME  MEASUREMENT 


Several  issues  emerge  from  a survey  of  the  outcome  measures  used 
in  career  counseling  research.  These  issues  are  neither  new  nor  unique 
to  career  counseling  research.  In  fact,  Williamson  and  Bordin  (1941) 
touched  on  many  of  the  same  problems  facing  career  counseling  researchers 
today  in  their  classic  article,  which  critiques  the  methodology  of  voca- 
tional and  educational  counseling  research  over  35  years  ago.  Problems 

encountered  in  outcome  research  in  psychotherapy  often  have  their  counter-  , 

parts  in  outcome  research  in  career  counseling  (Bergin  & Garfield,  1971, 

Meltzoff  6 Kornreich,  1970). 

These  various  problem  areas  are  interrelated.  Questions  of  validity, 
for  example,  are  linked  to  the  criterion  used,  whereas  decisions  concern- 
ing the  instruments  to  be  used  depend  on  the  operational  definitions  the 
researcher  chooses  for  the  constructs  to  be  measured.  The  issues  in  this 
section  relate  to  criteria,  instruments,  and  design  and  analysis.  Most 
problems  that  the  career  counseling  outcome  researcher  must  resolve  can 
be  subsumed  under  these  general  categories. 

Criteria  of  Career  Counseling  Outcome 

The  criterion  issue  has  perpetually  bedeviled  counseling  research, 
and  there  are  several  aspects  of  the  criterion  problem  to  consider.  One 
aspect  concerns  selection  of  immediate  or  ultimate  criteria.  Another 
facet  of  the  criterion  issue  relates  to  the  use  of  change  as  a criterion. 

In  addition,  there  are  infrequently  used  criterion  measures  that  deserve 
special  attention. 

Immediate  vs.  Ultimate  Criteria.  One  aspect  of  the  criterion  debate 
has  involved  the  relative  merits  of  immediate  and  ultimate  criteria. 

Gonyea  (1962)  pointed  out  that  the  attrition  problem  conplicates  the  col- 
lection of  long-range  criteria  data.  Because  the  attrition  of  subjects 
is  probably  selective,  sampling  problems  become  even  more  complex.  In 
addition,  the  longer  the  time  since  concluding  the  counseling,  the  greater 
the  probability  that  factors  other  than  the  counseling  experience  have 
affected  the  outcome.  The  effect  of  other  factors  may  account  for 
research  evidence  that  short-term  and  long-term  measures  of  the  same 
variable  do  not  correlate  very  highly.  Hewer  (1966),  for  example,  used 
the  current  occupation  of  her  subjects  as  the  criterion  of  appropriate- 
ness in  a followup  of  an  earlier  study  (Hewer,  1959).  She  compared  the 
earlier  judges'  ratings  of  appropriateness  with  the  occupations  the  sub- 
jects pursued  some  8 years  later  and  concluded  there  was  "no  relation- 
ship between  the  judges'  ratings  of  realism  of  vocational  choice  and 
realism  as  determined  by  similarity  of  the  original  choice  and  current 
employment"  (Hewer,  1966,  p.  292).  The  question  posed  by  Hewer's  re- 
search is  whether  the  lack  of  correlation  between  the  two  measures  of 
appropriateness  was  caused  by  the  lack  of  validity  of  judges'  ratings 
or  by  the  various  influences  on  the  subjects  during  the  years  between 
the  original  and  followup  studies. 
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Another  facet  of  immediate  versus  ultimate  criteria  has  been  sug- 
gested by  Katz  (1975),  who  pointed  out  that  we  do  not  require  long-term 
payoffs  for  competencies  such  as  typing,  reading  comprehension,  woodwork- 
ing, or  social  studies.  Accordingly,  Katz  questioned  the  need  for  assess- 
ment of  long  range,  real-life  outcomes  for  career  decisionmaking.  If  it 
is  agreed  that  career  decisionmaking  is  a desirable  skill,  Katz'  argument 
would  hold  that  the  researcher  determines  whether  the  skill  has  been 
acquired,  not  whether  the  skill  predicted  a long-range  criterion  such 
as  job  success.  Myers  (1971)  came  to  a similar  conclusion  from  his  con- 
ceptualization of  career  development  as  a series  of  choice  points,  with 
each  choice  influencing  the  one  that  comes  after  it.  Myers  argued  that 
it  is  unlikely  that  short-term  career  counseling  can  have  much  measur- 
able long-term  effect,  considering  the  far  greater  effect  each  choice 
actually  made  will  have  on  subsequent  choices. 

Change  as  a Criterion.  Another  aspect  of  the  criterion  problem 
relates  to  using  change  as  an  outcome  measure.  Some  investigators  have 
used  changes  in  career  choice  as  outcome  measures  (Anderson  & Binnie, 

1971)  or  changes  in  the  number  of  career  options  being  considered 
(Cooper,  1976;  Krivatsy  and  Magoon,  1976;  Mencke  & Cochran,  1974;  Zener 
and  Schnuelle,  1976).  It  is  possible,  though,  that  change  per  se  is  not 
necessarily  a desirable  outcome.  If,  in  changing,  the  client  ends  up 
with  a less  realistic  career  choice  than  he  or  she  had  before,  change 
is  undesirable.  In  like  manner,  an  increase  in  the  number  of  career 
options  might  be  a positive  outcome  for  clients  who  are  in  the  explora- 
tory stage  of  career  decisionmaking  whereas  a decrease  in  career  options 
would  be  advantageous  for  clients  in  a later  stage.  A similar  argument 
would  hold  for  other  variables,  such  as  certainty  of  career  choice. 

Higher  certainty  for  an  unrealistic  career  choice  might  be  less  desir- 
able than  lower  certainty  for  a realistic  choice.  Change,  even  if  its 
direction  is  predicted,  would  thus  appear  to  have  shortcomings  as  a de- 
pendent variable.  The  indication  of  such  considerations,  then,  is  that 
change  criteria  should  differ  from  client  to  client.  Some  authorities, 
e.g. , Bergin  (1971)  and  Krumboltz  (1966),  have  already  recommended  that 
the  goals  of  counseling  or  therapy  should  be  unique  to  each  client. 

Neglected  Criteria.  There  are  other  infrequently  used  criteria  that 
seem  to  have  advantages  as  outcome  measures  of  career  counseling.  Such 
criteria  include  instrumental  behaviors  and  cost  analysis. 

To  date,  almost  all  the  instrumental  behaviors  used  as  dependent 
variables  for  career  counseling  have  been  measures  of  some  kind  of  in- 
formation seeking.  There  is  no  reason,  however,  why  other  instrumental 
behaviors  could  not  be  used.  For  example,  did  the  client  obtain  a job 
(Thoresen  et  al. , 1967)  or  enter  a training  program?  In  the  Army  context, 
did  the  officer  actually  con$>lete  a career  plan?  Did  the  enlisted  woman 
register  for  and  conqplete  a civilian  education  course  that  would  enhance 
her  chances  for  promotion? 
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Another  infrequently  used  measure  is  cost-benefit  analysis.  With 
this  outcome  measure,  an  assessment  is  made  of  the  effectiveness  of  the 
career  counseling  treatment  and  of  its  cost.  Given  two  equally  effec- 
tive modes  of  test  interpretation,  for  example,  the  one  that  is  avail-  t 

able  at  lower  cost  probably  will  be  chosen.  On  the  other  hand,  a less  * 

expensive  approach  may  not  be  selected  as  the  treatment  of  choice  if 
another,  more  costly,  method  is  of  sufficiently  greater  effectiveness. 

Historically,  cost-benefit  techniques  have  been  used  primarily  by 
economists  in  evaluating  social  change  and  the  relative  desirability  of 
different  social  policies  (Rothenberg,  1975).  With  deflated  financial 
resources  of  government  agencies  and  other  institutions,  however,  the 
goal  of  such  analyses  in  evaluation  research  is  "to  determine  that 
strategy  or  combination  of  strategies  that  maximizes  the  desired  result 
for  any  particular  resource  or  budget  constraint"  (Levin,  1975,  p.  89). 

Since  accountability  is  of  increasing  importance  to  organizations  and 
institutions,  cost-benefit  analyses  are  likely  to  become  more  frequent 
in  career  counseling  outcome  research. 

The  advantages  of  some  of  these  less  frequently  used  outcome 
variables  in  terms  of  nonreactive  measurement  will  be  discussed  later. 

The  point  here  is  that  certain  criteria  not  extensively  used  hereto- 
fore— e.g.,  cost-benefit  analysis  or  archival  data  on  career  informa- 
tion seeking — might  be  considered  more  frequently  by  career  counseling 
researchers  as  supplementary  measures  to  more  traditional  criteria. 


Instruments 

After  the  criteria  of  career  counseling  outcome  have  been  chosen, 
instruments  must  be  selected  or  constructed.  Several  problems  may,  need 
to  be  resolved  in  making  decisions  about  instruments;  these  concerns 
involve  operational  definitions,  the  type  of  instrument,  and  the 
reliability  and  validity  question. 

Operational  Definitions.  Zytowski  and  Betz  (1972)  have  argued 
strongly  for  greater  care  in  measurement  in  counseling  research.  These 
authors  have  pointed  out  the  need  for  clear,  precise  definitions  of  the 
constructs  to  be  measured.  One  of  the  fuzzy  areas  of  career  counseling 
outcome  research  concerns  the  definition  of  career  "choice"  or  "prefer- 
ence." Many  reports  of  outcome  research  do  not  make  clear  what  the 
subject  was  asked  to  do  in  specifying  his  or  her  career  choice.  Crites 
(1969)  has  presented  a comprehensive  discussion  of  this  particular  defi- 
nitional problem  (pp.  127-135).  In  his  discussion,  Crites  reviewed  the 
ways  in  which  career  choice  has  been  defined,  made  a critical  analysis 
of  these  definitions,  developed  an  operational  definition  of  choice,  and 
enumerated  the  necessary  conditions  for  the  expression  of  a career  choice. 
Crites'  ( 1969)  text  also  contains  a section  on  measures  of  vocational 
choice  in  which  the  author  has  described  previously  used  instruments  (pp. 
135-148).  Crites'  treatment  of  the  problem  of  defining  and  measuring 
career  choice  is  an  example  of  the  type  of  approach  that  Zytowski  and 
Betz  (1972)  have  called  for  with  respect  to  career  counseling  constructs. 
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defined,  the  researcher  mast  decide  which  type  of  instrument  is  most 
suitable  for  the  objective.  As  shown  in  Tables  1-8,  most  measures  in 
career  counseling  outcome  research  have  involved  self-report.  Zytowski 
and  Betz  ( 1972)  reported  that  two-thirds  of  the  instruments  they  reviewed 
were  self-report  measures.  Most  of  the  remainder  required  the  quantifi- 
cation of  judgments  about  other  people  or  the  counseling  process. 

Self-reports  have  been  obtained  by  questionnaires,  interviews,  Q-sort 
techniques,  or  tests.  Judges'  ratings  have  also  been  extensively  used, 
especially  for  measures  of  realism  and  self-knowledge.  Even  when  judges' 
ratings  comprised  the  criterion,  self-report  measures  (such  as  test  results 
or  expressed  career  choice)  were  usually  used  in  making  the  judgments. 

For  the  same  criterion,  several  different  types  of  instruments  could 
be  used.  If  the  dependent  variable  involves  career  information  seeking, 
for  example,  data  can  be  obtained  through  a questionnaire,  e.g.,  Zener 
and  Schnuelle  (1976);  an  interview,  e.g.,  Krumboltz  and  Thoresen  (1964); 
or  counseling  center  records,  e.g..  Cooper  (1976).  The  ease,  precision, 
and  reactivity  of  the  various  types  of  data  collection  must  be  weighed  in 
selecting  the  best  instrument  for  any  given  situation.  Although  a ques- 
tionnaire may  be  easily  administered  and  objectively  scored,  it  may  lack 
the  flexibility  of  an  interview  in  following  up  on  aspects  of  the  research 
not  anticipated  beforehand.  Counseling  center  records  may  provide  the 
means  for  an  unobtrusive  measure  of  career  information  seeking,  but  simi- 
lar records  may  be  unavailable  in  other  settings. 

To  a large  extent,  dependent  variables  used  in  career  counseling 
research  have  tended  to  be  "reactive"  measures  (Webb  et  al. , 1966).  Thus, 
the  measurement  process  itself  may  influence  the  outcome  of  a career  coun- 
seling experiment.  Such  an  effect  would  be  especially  likely  in  pretest- 
posttest  designs,  commonly  used  in  career  counseling  outcome  research. 
Although  real  change  may  occur,  it  may  be  because  of  the  measurement 
activity  rather  than  the  treatment.  Randomization  and  the  inclusion  of 
control  groups,  two  common  techniques  used  to  assure  "internal"  validity 
(Campbell  & Stanley,  1966),  may  be  ineffective  when  reactive  measures 
are  used. 

Considering  the  unknown  and  possibly  sizable  effects  of  reactive 
measurement,  it  is  surprising  that  so  few  career  counseling  researchers 
have  not  attempted  to  assess  or  control  for  such  effects.  One  way  of 
determining  the  effect  of  a pretest,  for  example,  would  be  to  employ  a 
Solomon  four-group  design  (Campbell  & Stanley,  1966,  pp.  24-25).  A 
posttest-only  control  group  design  will  also  eliminate  pretest  effects  by 
eliminating  the  pretest.  With  respect  to  this  latter  design,  Campbell  and 
Stanley  (1966)  commented  that  although  the  pretest  is  "a  concept  deeply 
embedded  in  the  thinking  of  research  workers  in  education  and  psychology, 
it  is  not  actually  essential  to  true  experimental  designs"  (p.  25).  These 
authors  have  also  noted  that  although  the  Solomon  four-group  design  is 
preferable,  its  advantages  may  not  outweigh  the  doubled  effort  involved 
unless  randomization  is  not  possible. 
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An  inspection  of  Tables  1-8  reveals  that  only  two  studies  used  non- 
reactive outcome  measures.  Krivatsy  and  Magoon  (1976)  analyzed  the  costs 
of  materials  and  personnel  time  to  evaluate  the  relative  expense  of  three 
different  career  counseling  treatments,  and  Cooper  ( 1976)  recorded  whether 
or  not  subjects  returned  to  the  counseling  center  for  career  information. 
Dependent  variables  of  the  types  just  described  are  examples  of  measures 
both  objective  and  nonreactive.  It  is  my  opinion  that  a larger  propor- 
tion of  career  counseling  outcome  measures  should  be  objective  and  non- 
reactive, especially  in  cases  where  randomization  cannot  be  achieved  or 
the  comparison  group  is  not  a true  control  group. 

For  a comprehensive  discussion  of  the  hazards  of  reactive  measure- 
ment and  a compendium  of  unobtrusive  measures,  see  Webb  et  al.  (1966). 
Weinstein's  (1975)  chapter  on  using  information  systems  to  evaluate  mental 
hygiene  programs  also  contains  information  pertinent  to  assessing  career 
counseling  outcomes.  Researchers  charged  with  the  evaluation  of  counsel- 
ing center  operations  may  find  Weinstein's  paper  of  particular  interest. 

i 

Reliability  and  Validity.  Another  problem  the  career  counseling 
researcher  must  consider  in  devising  or  selecting  instruments  relates  to 
their  reliability  and  validity.  Zytowski  and  Betz  (1972),  in  their  re- 
view of  measurement  in  counseling  research,  documented  the  inadequacy  of 
the  reliability  and  validity  data  that  have  been  reported  in  the  litera- 
ture. The  career  counseling  outcome  research  that  has  been  surveyed  in 
this  paper  has  tended  to  be  remiss  in  presenting  reliability  and  validity 
data.  Exceptions  to  this  generalization  are  criteria  involving  judges' 
ratings  and  career  information  seeking.  About  half  the  studies  using 
judges'  ratings  presented  data  on  interrater  reliability.  Rogers  (1954) 
reported  test-retest  reliability  of  the  instrument  he  devised  for  self- 
estimates of  interests  and  aptitudes,  and  Linden  et  al.  (1965)  provided 
both  reliability  and  validity  data  on  their  Counseling  Evaluation 
Inventory. 

Zytowski  and  Betz  (1972)  concluded  that  counseling  research  (either 
process  or  outcome)  should  not  be  published  without  "some  minimal  indica- 
tions of  the  reliability  and  validity  of  the  instrument  on  which  the  re- 
search is  based"  (p.  78).  Specifically,  they  recommended  that  researchers 
report  the  results  of  any  previous  reliability  studies,  estimates  of  in- 
ternal consistency,  and  test-retest  reliability  (where  indicated).  With 
respect  to  the  more  difficult  problem  of  validity,  Zytowski  and  Betz  sug- 
gested that  construct  validity  is  perhaps  the  most  feasible  type.  The 
researcher  should  first  write  a descriptive  statement  that  clearly  and 
precisely  defines  the  construct  and  then  generate  validity  data  through 
research  on  group  differences,  correlation  with  other  variables,  treat- 
ment effects,  and  the  like.  For  exanple,  realism  of  alternate  specialty 
preference  could  be  defined  in  terms  of  a choice  that  is  consonant  with 
an  officer's  educational  level,  undergraduate  (or  graduate)  major,  previ- 
ous assignments,  and  primary  specialty.  Research  then  could  be  designed 
to  determine  whether  degree  of  realism  of  alternate  specialty  preference 
differentiated  officers  who  leave  the  Army  early  in  their  careers  from 
those  who  leave  it  later  (group  differences).  Other  studies  could 
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investigate  such  questions  as  the  relationship  between  realism  of  the  pref- 
erences and  later  satisfaction  in  those  alternate  specialties  (correlation 
with  other  variables)  or  the  effects  of  a computer-aided  career  planning 
intervention  on  the  realism  of  expressed  preferences  (treatment  effects). 


and  Analysis 


Researchers  face  other  difficulties  relating  to  experimental  design. 
Some  of  the  problems  may  be  particularly  troublesome  to  the  person  engaged 
in  career  counseling  outcome  research,  because  such  research  frequently 
involves  real  clients,  real  counselors,  and  real  counseling  environments. 

In  all  field  research,  the  researcher  exercises  less  control  over  the 
research  design  than  is  usually  the  case  in  a laboratory  setting.  Lesser 
control  may  jeopardize  both  the  internal  and  external  validity  of  the  study. 
Design  concerns  considered  here  include  local  control,  assignment  of  sub- 
jects, and  comparison  groups.  Design  type  is  considered  in  relation  to 
these  concerns,  and  some  associated  analysis  problems  are  noted. 


Local  Control.  Volsky  et  al.  ( 1965)  have  used  the  term  "local  control" 
for  procedures  that  either  eliminate  factors  contaminating  the  dependent 
variable  or  incorporate  the  factors  into  the  research  design.  Volsky  et  al. 
called  these  techniques  "controlling  out"  and  "controlling  in."  An  example 
of  the  first  type  would  be  restricting  one's  sample  to  self -ref erred  cli- 
ents, thus  "controlling  out"  the  contaminating  influence  of  persons  who 
have  been  required  to  come  for  counseling.  The  second  type  would  be  il- 
lustrated by  incorporating  into  the  design  different  client  characteristics 
such  as  gender  or  level  of  career  maturity.  When  such  controls  are  built 
into  the  research  design,  it  is  possible  to  detect  main  effects  and  inter- 
actions due  to  these  variables. 

Assignment  of  Subjects.  Randomization  is  the  sine  qua  non  of  ex- 
perimental research.  Assigning  subjects  randomly  does  not  mean  that  the 
groups  are  equated  on  pertinent  variables.  However,  it  does  assure  that 
the  risk  of  significant  imbalance  is  the  same  as  the  level  of  risk  as- 
sumed by  the  statistical  techniques  employed. 


Desirable  as  randomization  is,  however,  it  cannot  always  be  attained. 
Particularly  in  fieldwork,  randomization  nay  be  impractical  if  not  impos- 
sible. A researcher  may  have  to  work  with  intact  groups  such  as  orienta- 
tion classes,  for  example,  selected  by  the  student  subjects  for  nonrandom 
reasons  such  as  time  of  day.  Even  when  clients  are  randomly  assigned  to 
career  counseling  groups,  the  randomization  process  may  be  preempted  by 
the  restrictions  imposed  by  the  subjects'  course  schedules. 

When  randomization  is  imperfect,  the  risk  of  an  interaction  be- 
tween subject  selection  and  treatment  is  increased.  Kelley  et  al.  ( 1970) 
assessed  73  reports  of  counseling  research  for  sources  of  invalidity  and 
found  that  a selection-treatment  interaction  was  a "definite  weakness"  in 
61.6%  of  the  studies  and  a "probable  weakness"  in  the  remaining  38.4%. 
Although  the  investigations  reviewed  by  Kelley  et  al.  were  not  limited 
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to  career  counseling  and  included  process  as  well  as  outcome  research, 
his  results  illustrate  the  prevalence  of  inadequate  randomization. 

For  a more  detailed  discussion  of  randomization  and  its  associated 
problems  for  counseling  research,  the  reader  is  referred  to  Volsky  et  al. 
(1965,  chapter  IV). 

Comparison  Groups.  One  major  problem  in  all  counseling  and  therapy 
research  lies  in  the  difficulty  of  obtaining  appropriate  comparison  groups. 
Usually,  the  comparison  is  made  between  an  "experimental"  group  that  re- 
ceived the  treatment  or  intervention  and  a "control"  group  that  did  not. 

To  deny  treatment  to  individuals  who  want  it  in  order  to  obtain  a control 
group  may  not  be  ethically  defensible.  The  solution  to  this  ethical 
dilemma  in  career  counseling  research  has  often  been  to  use  a "wait-con- 
trol" group.  After  the  subjects  are  randomly  assigned  to  two  groups,  one 
group  of  subjects  is  given  the  counseling  intervention  immediately,  and 
treatment  for  the  control  group  is  postponed  until  after  posttest 
measures  are  made  on  both  groups. 

Unfortunately,  denying  or  postponing  treatment  does  not  necessarily 
preclude  the  control  subjects'  receiving  help  from  other  sources.  People 
who  are  motivated  to  seek  career  counseling  in  the'  first  place  may  also 
be  motivated  to  seek  help  from  other  sources,  rather  than  wait  until  the 
termination  of  counseling  for  the  experimental  group.  In  Campbell  and 
Stanley's  (1966)  terms,  "history"  as  a source  of  invalidity  may  contami- 
nate the  control  group  or  individuals  in  the  control  group.  Some  event 
other  than  the  experimental  treatment  occurs  that  may  have  some  of  the 
same  effects  a3  the  treatment. 

In  a discussion  of  the  effects  of  withholding  counseling  services, 
Volsky  et  al.  (1965)  reported  a study  in  which  all  control  subjects  in 
the  experiment  refused  treatment  when  it  was  finally  available  to  them. 

This  situation  apparently  occurred  because  the  experimental  treatment 
ended  near  the  end  of  the  spring  term,  with  final  examinations  impending. 
Given  that  there  are  such  effects,  Volsky  et  al.  (1965)  asked,  "What  can 
be  used  as  the  placebo  comparable  in  all  dimensions  except  the  relevant 
treatment  variable  in  subsequent  experimentation?"  (p.  161).  The  authors 
concluded  that  the  best  that  can  be  done  in  terms  of  classical  experimen- 
tal methods  is  to  compare  treatment  methods,  using  random  assignment  of 
subjects  to  the  various  groups. 


Control  groups  may  be  less  necessary  in  career  counseling  outcome 
research  than  in  other  types  of  psychological  research.  Almost  all 
career  counseling  outcome  studies  using  a control  group  found  their 
experimental  groups  scored  higher  on  the  outcome  measures  than  did  the 
control  groups,  e.g.,  Hoyt  (1955),  Tipton  (1969),  and  Wright  (1963). 

In  fact,  this  superiority  of  career  counseling  over  none  is  about  the 
only  consistent  finding  of  outcome  research.  Thus,  the  suggestion  of 
Volsky  and  his  colleagues  to  compare  different  methods  of  counseling 
instead  of  comparing  experimental  and  control  groups  may  represent  a 
practical  course  for  the  career  counseling  researcher. 
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Type  of  Design.  Campbell  and  Stanley  (1966)  and  Cook  and  Campbell 
(1976)  have  presented  in  detail  the  various  types  of  pre-experimental, 
experimental,  and  quasi-experimental  designs  available  to  researchers  in 
the  behavioral  and  social  sciences.  Because  pretest-posttest  designs 
have  been  used  so  frequently  in  career  counseling  outcome  research,  the 
focus  of  this  discussion  will  be  on  the  pretest-posttest  design  and  some 
alternatives  to  it.  The  researcher’s  choice  of  design  and  type  of  data 
analysis  will  depend  to  a great  extent  on  whether  subjects  can  be  ran- 
domly assigned  to  treatments  and  on  the  type  of  comparison  group(s) 
available. 

A research  design  proposed  by  Crites  (1964)  includes  control  groups 
and  allows  the  researcher  to  assess  the  effects  (in  many  instances)  of 
pretesting  and  time,  both  of  which  are  possible  sources  of  confounding. 

In  his  description  of  this  design,  Crites  has  noted  possible  statistical 
tests  which  can  be  made  and  has  suggested  the  possible  significance  of 
results  in  terms  of  counseling  effects. 

Though  often  used,  pretest-posttest  designs  involve  other  hazards 
for  outcome  research  (unless  controlled  for  as  in  the  Crites  design). 
Administering  a pretest  may  sensitize  subjects  to  the  treatment  they  are 
to  receive  and  may  also  produce  a practice  effect.  Accordingly,  Campbell 
and  Stanley  ( 1966)  have  pointed  out  the  assets  of  the  posttest-only  de- 
sign. Those  authors  suggested  that  the  advantages  of  the  Solomon  four- 
group  design  over  the  posttest-only  control  group  design  may  not  compen- 
sa^e  ^or  ^he  doubled  effort.  The  Solomon  four-group  design  permits  the 
investigator  to  assess  the  effects  of  the  pretest.  Nunnally  (1975)  has 
also  recommended  the  posttest-only  design,  characterizing  it  as  "the 
workhorse  of  evaluation  research"  (p.  123).  If  certain  conditions  apply, 
however,  a pretest-posttest  design  should  be  considered. 

These  conditions  are  that  the  (a)  nunber  of  subjects  in  each  cell  is 
small,  (b)  subjects  shew  a wide  variation  (large  standard  deviation)  on 
the  dependent  variable  compared  to  the  expected  effects  of  the  independent 
variable  (treatment),  and  (c)  pretest-posttest  correlation  of  the  depen- 
dent variable  is  expected  to  be  high.  When  these  three  conditions  apply 
and  it  is  reasonable  to  expect  pretest-treatment  interactions  and  pretest- 
P°8tt®st  interactions  to  be  slight,  Nunnally  has  suggested  using  a pretest- 
posttest  design. 

Although  the  career  counseling  outcome  researcher  often  may  be  able 
to  assign  subjects  randomly  to  different  treatments  and  to  a wait-control 
group,  sometimes  such  procedures  are  not  possible.  Intact  groups,  e.g. , 
orientation  classes,  may  have  to  receive  the  different  treatments  and  to 
serve  as  the  control  group(s).  When  intact  groups  must  be  used,  Nunnally 
(1975)  suggested  using  a control  group  and  obtaining  measures  on  the  de- 
pendent variable  before  and  after  the  counseling  treatment.  With  this 
type  of  quasi-experimental  design,  then,  pretest-posttest  measures  are 
needed. 


34 


! 

] 


Data  Analysis.  Whether  an  experimental  or  quasi-experimental  de- 
sign is  used  determines  the  type  of  analysis  to  be  used.  Nunnally  (1975) 
stated  that  the  statistical  analysis  of  data  in  true  experiments  is  gen- 
erally not  controversial;  but  such  is  not  the  case  for  methods  of  analyz- 
ing data  from  quasi-experimental  designs.  For  example,  Nunnally  chal- 
lenged the  practice  of  using  covariance  analysis  when  an  intact  group  is 
used  as  a control  group.  Instead,  the  author  suggested  using  a repeated- 
measures  analysis  of  variance  with  the  pretest-posttest  measures  as  a 
i within-sub jects  factor.  In  this  analysis,  the  focus  of  interest  would  be 

the  interaction  of  the  treatment  condition  with  the  comparison  groups. 
Nunnally  has  presented  examples  of  both  simple  and  complex  designs  of 
this  type.  He  also  suggested  using  a conservative  approach  for  any  post 
hoc  tests  and  recommended  obtaining  measures  of  strength  of  association 
to  help  the  researcher  decide  whether  differences  between  treatments  are 
meaningful  for  real  world  situations.  The  reader  is  referred  to  Nunnally 
(1975,  pp.  134-136)  for  recommendations  and  rationales. 


IMPLICATIONS  FOR  FUTURE  RESEARCH 

This  paper  has  surveyed  the  instruments  used  in  career  counseling 
outcome  research,  and  some  of  the  issues  raised  by  the  outcome  research 
have  been  considered.  What,  then,  are  the  implications  of  the  findings 
for  the  career  counseling  researcher?  The  intent  of  the  following  discus- 
sion is  to  focus  on  some  specific  recommendations  that  emerge  from  the 
shortcomings  of  career  counseling  outcome  research.  Most  suggestions 
were  made  previously  by  authors  critiquing  research  on  psychotherapy, 
counseling,  and  guidance.  For  clarity,  the  specific  points  to  consider 
will  be  grouped  as  they  relate  to  criteria,  instruments,  or  design  and 
analysis. 


Recommendations  for  Criteria 
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Use  Multiple  Criteria.  Don't  use  only  one  dependent  variable;  use 
several.  Career  counseling  outcomes  are  multidimensional;  one  measure 
is  not  adequate.  Also  avoid  using  all  the  same  types  of  measure.  For 
example,  include  behavioral  and  archival  data,  as  well  as  attitudinal 
variables. 


Use  Specific  Rather  Than  Global  Measures.  The  use  of  a global 
criterion  may  not  tap  the  underlying  dimensions.  Bergin  and  Garfield 
(1971)  stressed  the  superiority  of  specific  over  global  measures  and  also 
suggested  that  perhaps  individual  criteria  should  be  developed  for  each 
client.  Krumboltz  ( 1966)  also  argued  for  unique  goals  for  each  client. 
Career  counseling  outcome  research  has  not  incorporated  such  criteria 
into  research  designs  as  yet.  An  example  given  earlier  concerned  change 
as  a criterion.  An  increase  in  career  options  might  be  desirable  for 
one  client  and  a decrease  desirable  for  another,  depending  on  the 
client's  career  decisionmaking  stage. 
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Emphasize  Short-Term  Measures.  It  may  be  unrealistic  to  expect 
long-term  effects  from  the  minimal  interventions  used  in  most  career 
counseling  research  (Myers,  1971).  A well-designed,  large-scale,  longi- 
tudinal study  obviously  will  require  repeated  measures  over  a long  period 
of  time.  Because  most  career  counseling  outcome  research  is  not  of  this 
type,  short-term  measures  are  usually  called  for. 

Define  Constructs  Carefully.  Mitchell  et  al.  (1974)  expressed  the 
opinion  that  most  studies  on  career  decisionmaking  are  of  limited  use  be- 
cause constructs  and  instruments  are  not  comparable  across  the  various 
studies.  Zytowski  and  Betz  (1972)  also  stressed  the  need  for  more  care- 
ful definitions  of  constructs.  An  example  was  given  previously  from 
Crites  ( 1969)  that  illustrated  the  type  of  approach  needed. 


Recommendations  Relating  to  Instruments 

Use  Instruments  From  Previous  Research.  Obviously,  one  cannot  follow 
this  suggestion  if  no  instrument  exists  that  measures  the  construct  of  in- 
terest. All  too  often,  however,  researchers  "reinvent  the  wheel."  Con- 
structing a good  instrument  is  not  an  easy  task.  A larger  pool  of  sensi- 
tive and  reliable  instruments  would  be  available  if  each  researcher  built 
on  the  work  of  predecessors  in  the  area  of  instrumentation. 

Report  Reliability  and  Validity  Data  for  the  Instruments  You  Use.  Of 
the  studies  surveyed  in  this  paper,  about  one-third  presented  data  on  reli- 
ability and  one-sixth  on  validity.  A major  point  made  by  Zytowski  and  Betz 
(1972)  was  that  information  on  reliability  and  validity  should  be  reported 
routinely  on  all  instruments  used  in  counseling  research.  These  authors 
also  found  that  such  data  typically  are  not  included  in  reports  of  counsel- 
ing research.  If  a researcher  uses  an  already-developed  instrument  for 
which  reliability  and  validity  have  been  established,  considerable  time 
and  effort  can  be  saved.  An  investigator  should  be  aware,  however,  that 
in  order  to  obtain  comparable  results,  it  is  necessary  to  use  the  instru- 
ment in  the  same  fashion  that  the  original  researcher  used  it.  A number 
of  career  counseling  outcome  studies  have  used  only  portions  of  standard- 
ized instruments.  Unless  the  reliability  and  validity  of  the  adapted 
instrument  can  be  established,  there  is  no  assurance  that  it  measures 
the  same  construct  or  is  as  reliable  as  the  original  instrument. 

Use  Objective  Rather  Than  Subjective  Measures.  In  general,  ratings 
by  subjects  or  judges  are  not  considered  as  valid  measures  as  are  more 
objective  types  of  assessment  devices.  Not  one  of  the  "good"  studies 
Meltzoff  and  Kornreich  (1970)  surveyed  used  judges'  or  patients'  ratings 
of  therapy  outcome.  Even  if  the  same  instrument  (rating  form)  is  used, 
results  may  not  be  reliable  if  different  raters  are  used  across  studies. 
Judges'  ratings  have  been  used  extensively  for  certain  variables,  such 
as  appropriateness  of  career  choice.  A more  objective  method  of  assess- 
ing appropriateness  is  conparing  the  Holland  code  from  a test  (such  as 
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the  VPI , the  SDS,  or  the  Strong-Campbell  Interest  Inventory)  with  the 
Holland  code  of  the  subject's  expressed  career  choice,  e.g.,  Mencke 
and  Cochran  (1974),  Zener  and  Schnuelle  (1976). 
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Make  Use  of  Nonreactive  Measures  Whenever  Possible.  Because  non- 
reactive measurement  has  so  many  merits  (Webb  et  al.,  1966),  it  is  some- 
what surprising  that  so  few  career  counseling  outcome  studies  have  used 
unobtrusive  measures.  However,  such  measures  are  much  more  difficult  to 
devise  and  may  require  considerably  more  ingenuity  on  the  part  of  the 
researcher.  Archival  data  such  as  that  used  by  Cooper  (1976)  and  the 
cost  benefit  analysis  technique  employed  by  Krivatsy  and  Magoon  (1976) 
are  nonreactive  outcome  criteria  already  in  the  career  counseling 
research  literature. 
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Recommendations  Concerning  Design  and  Analysis 

Assign  Subjects  Randomly  to  Treatment  and  Control  Groups.  If  ran- 
domization is  used,  the  investigator  can  assume  that  no  systematic  bias 
was  introduced  into  the  composition  of  the  various  groups.  Thus,  if 
random  assignment  is  possible,  the  researcher  can  place  greater  confi- 
dence in  the  internal  validity  of  his  or  her  study.  If  intact  groups 
must  be  used,  employ  a pretest-posttest  design  and  analyze  the  data  by 
a repeated  measures  analysis  of  variance  (Nunnally,  1975). 


Include  a Control  Group  in  the  Research  Design.  A control  group  is 
needed  to  establish  the  internal  validity  of  a study.  The  ethical  ques- 
tion on  denial  of  treatment  can  be  resolved  by  using  a wait-control  group. 
This  group  will  receive  career  counseling  after  treatment  has  been  con- 
cluded for  the  experimental  subjects.  If  no  control  group  is  possible, 
compare  different  treatments. 

Estimate  Sample  Size  Needed.  As  Bailey  (1971)  has  pointed  out,  the 
use  of  more  subjects  than  are  needed  to  detect  meaningful  differences  is 
"a  waste  of  resources"  (p.  323).  Career  counseling  researchers  have 
limited  time  as  well  as  money  for  tests  and  other  materials.  Many  re- 
searchers, thus,  omit  the  step  of  calculating  the  size  of  the  sample 
they  need.  One  reason  for  neglect  of  this  procedure  is  the  difficulty 
in  estimating  how  large  a difference  would  have  to  exist  in  order  for  it 
to  matter.  Such  judgments  require  the  researcher's  subjective  evaluation 
of  the  role  of  factors  such  as  the  kind  of  outcome  measure,  the  charac- 
teristics of  the  sample,  and  the  level  of  effort  required  by  the  treatment. 
More  detailed  discussion  of  the  problem  of  estimating  sample  size  and  the 
methods  to  use  can  be  found  in  sources  such  as  Bailey  (1971)  and  Winer 
(1971). 
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SUMMARY 


This  paper  reviewed  measures  previously  used  in  career  counseling 
outcome  research.  The  purpose  of  the  review  was  to  provide  background 
for  the  construction  of  instruments  to  be  used  in  evaluating  a computer- 
based  career  information  and  planning  system  for  Army  officers. 

Some  of  the  general  considerations  relating  to  career  counseling 
outcome  research  were  the  types  of  behaviors  to  be  measured,  where  and 
when  the  assessment  would  take  place,  the  availability  «f  valid  and 
reliable  instrumentation,  and  the  objectives  of  the  research  effort. 
Previously  used  career  counseling  outcome  measures  were  surveyed  using 
a taxonomy  from  Myers  (1971),  which  was  expanded  slighted  to  incorpo- 
rate additional  types  of  measures.  Each  kind  of  measure  was  described, 
a typical  example  of  it  was  presented,  and  any  variants  of  that  type  were 
noted.  For  the  reader's  convenience,  the  various  studies  were  summarized 
in  eight  tables  which  corresponded  to  the  eight  major  categories  of  out- 
come measures. 

From  this  survey  of  career  counseling  outcome  measures,  a number 
of  interrelated  issues  emerged.  These  issues  were  discussed  in  terms  of 
their  relationship  to  criteria,  instruments,  or  design  and  analysis. 

The  implications  of  these  issues  for  future  research  were  incorporated 
into  a series  of  recommendations  for  the  career  counseling  outcome 
research. 
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1 USA  Air  Def  Bd.  Ft  Bliss,  ATTN:  STEBD-PO 
1 USA  Cmd  ft  General  Stf  College,  Ft  Leavenworth,  ATTN:  Lib 
1 USA  Cmd  ft  General  Stf  College,  Ft  Leavenworth,  ATTN:  ATSW-SE-L 
1 USA  Cmd  ft  General  Stf  College,  Ft  Leavenworth,  ATTN:  Ed  Advisor 
1 USA  Combined  Arms  Cmbt  Dev  Act.  Ft  Leavenworth,  ATTN:  DepCdr 
1 USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth,  ATTN:  CCS 
1 USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth,  ATTN:  ATCASA 
1 USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth,  ATTN:  ATCACO-E 
1 USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth.  ATTN:  ATCACC-CI 
1 USAECOM,  Night  Vision  Lab.  Ft  Belvoir,  ATTN:  AMSEL-NV-SD 

3 USA  Computer  Sys  Cmd,  Ft  Belvoir,  ATTN:  Tech  Library 
1 USAMERDC.  Ft  Belvoir.  ATTN:  STSFB-DQ 

1 USA  Eng  Sch,  Ft  Belvoir,  ATTN:  Library 
1 USA  Topographic  Lab,  Ft  Belvoir,  ATTN:  ETL-TD— $ 

1 USA  Topographic  Lab,  Ft  Belvoir,  ATTN:  STINFO  Canter 
1 USA  Topographic  Lab,  Ft  Belvoir,  ATTN:  ETL-GSL 
1 USA  Intelligence  Ctr  ft  Sch,  Ft  Huachuca,  ATTN:  CTD— MS 
1 USA  Intelligence  Ctr  ft  Sch,  Ft  Huachuca,  ATTN:  ATS-CTD-MS 
1 USA  Intelligence  Ctr  ft  Sch,  Ft  Huachuca,  ATTN:  ATSI-TE 
1 USA  Intelligence  Ctr  ft  Sch,  Ft  Huachuca,  ATTN:  ATSI— TEX-GS 
1 USA  Intelligence  Ctr  ft  Sch,  Ft  Huachuca,  ATTN:  ATSI-CTS-OR 
1 USA  Intelligence  Ctr  ft  Sch,  Ft  Huachuca,  ATTN:  ATSI-CTD-DT 
1 USA  Intelligence  Ctr  ft  Sch,  Ft  Huachuca,  ATTN:  ATSI— CTD-CS 
1 USA  Intelligence  Ctr  ft  Sch,  Ft  Huachuca,  ATTN:  DAS/SRD 
1 USA  Intelligence  Ctr  ft  Sch,  Ft  Huachuca,  ATTN:  ATSI-TEM 
1 USA  Intelligence  Ctr  ft  Sch,  Ft  Huachuca,  ATTN:  Library 

1 CDR,  HQ  Ft  Huachuca.  ATTN:  Tech  Ref  Div 

2 CDR,  USA  Electronic  Prvg  Grd,  ATTN:  STEEP-MT-S 
1 CDR.  Project  MASSTER,  ATTN:  Tech  Info  Canter 

1 Hq  MASSTER.  USATRADOC.  LNO 
1 Reaearch  Institute,  HQ  MASSTER,  Ft  Hood 
1 USA  Recruiting  Cmd,  Ft  Sherdian,  ATTN:  USARCPM-P 
1 Senior  Army  Adv.,  USAFAGOD/TAC,  Elgin  AF  Aux  Fid  No.  0 
1 HQ  USARPAC,  DCSPER,  APO  SF  96668,  ATTN:  GPPE-SE 
1 Stimson  Lib.  Academy  of  Health  Sciences,  Ft  Sam  Houston 
1 Marine  Corps  Inst..  ATTN:  Dean -MCI 
1 HQUSMC,  Commandant,  ATTN:  Code  MTMT  51 

1 HQUSMC.  Commandant,  ATTN:  Code  MPl-20 

2 USCG  Academy,  New  London,  ATTN:  Admission 
2 USCG  Academy,  New  London,  ATTN:  Library 

1 USCG  Training  Ctr,  NY.  ATTN:  CO 
1 USCG  Training  Ctr,  NY,  ATTN:  Educ  Svc  Ofc 
1 USCG.  Psychol  Res  Br.  DC.  ATTN:  GP  1/62 
1 HQ  Mid-Range  Br,  MC  Det,  Quantico.  ATTN:  PftS  Div 
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1  US  Marine  Corps  Liaition  Ofc,  AMC,  Alexandria,  ATTN:  AMCGS— F 
1 USATRADOC,  Ft  Monroe,  ATTN:  ATRO--ED 
6 USATRADOC.  Ft  Monroe.  ATTN:  ATPR-AO 
1 USATRADOC,  Ft  Monroe.  ATTN:  ATTS-EA 

1 USA  Forces  Cmd,  Ft  McPherson,  ATTN:  Library 

2 USA  Aviation  Tast  Bd.  Ft  Rucker,  ATTN:  STEBG-PO 

1 L'SA  Aflcy  for  Aviation  Safety,  Ft  Rucker,  ATTN:  Library 
1 USA  Agcy  for  Aviation  Safety,  Ft  Rucker,  ATTN:  Educ  Advisor 
1 USA  Aviation  Sch.  Ft  Rucker.  ATTN:  PO  Drawer  0 

1 HOUSA  Aviation  Sys  Cmd.  St  Louis,  ATTN:  AMSAV-ZOR 

2 USA  Aviation  Sys  Teat  Act . Edwards  AFB.  ATTN:  SAVTE-T 
1 USA  Air  Def  Sch,  Ft  Bliss.  ATTN:  ATSA  TEM 

1 USA  Air  Mobility  Rsch  & Dev  Lab,  Moffett  Fid,  ATTN:  SAVDL-AS 
1 USA  Aviation  Sch,  Res  Tng  Mgt,  Ft  Rucker,  ATTN:  ATST-T-RTM 
1 USA  Aviation  Sch,  CO,  Ft  Rucker,  ATTN:  ATST-D-A 
1 HO,  DARCOM."  Alexandria.  ATTN:  AMXCO-fL 
1 HQ.  DARCOM.  Alexandria.  ATTN:  COR 
1 US  Military  Academy,  West  Point.  ATTN:  Serials  Unit 
t US  Military  Academy,  West  Point,  ATTN:  Ofc  of  Milt  Ldrshp 
1 US  Military  Academy,  West  Point,  ATTN:  MAOR 
1 USA  Standardization  Gp,  UK,  FPO  NY,  ATTN:  MASE-GC 
1 Ofc  of  Naval  Rsch.  Arlington,  ATTN:  Coda  462 

3 Ofc  of  Naval  Rsch,  Arlington,  ATTN:  Coda  468 
t Ofc  of  Naval  Rsch,  Arlington,  ATTN:  Coda  460 
1 Ofc  of  Naval  Rsch,  Arlington,  ATTN:  Code  441 

1 Naval  Aerospc  Med  Res  Lab,  Pensacola,  ATTN:  Acous  Sch  ON 
1 Naval  Aerospc  Med  Res  Lab,  Pensacola,  ATTN : Coda  L51 
1 Naval  Aerospc  Med  Res  Lab,  Pensacola,  ATTN:  Coda  L6 
1 Chief  of  NavPers,  ATTN:  Pers-OR 
1 NAVAIRSTA,  Norfolk,  ATTN:  Safety  Ctr 
1 Nav  Oceanographic,  DC,  ATTN:  Code  6261,  Charti  & Tech 
1 Center  of  Naval  Anal,  ATTN:  Doc  Ctr 
1 NavAirSysCom,  ATTN:  AIR-6313C 
1 Nav  BuMed.  ATTN:  713 
1 NavHalicoptarSubSqua  2,  FPO  SF  96601 
1 AFHRL  (FT)  William  AFB 
1 AFHRL  (TT>  Lowry  AFB 

1 AFHRL  (AS)  WPAFB,  OH 

2 AFHRL  (DOJZ)  Brooks  AFB 

1 AFHRL  (DOJNI  Lackland  AFB 
1 HQUSAF  (INYSO) 

1 HOUSAF  (DPXXA) 

1 AFVTG  (RDI  Randolph  AFB 

3 AMRL  (HE)  WPAFB,  OH 

2 AF  Inst  of  Tech,  WPAFB,  OH,  ATTN:  ENE/SL 
1 ATC  (XPTDI  Randolph  AFB 

1 USAF  AeroMed  Lib,  Brooks  AFB  (SUL-41,  ATTN:  OOC  SEC 
1 AFOSR  (NLI,  Arlington 

1 AF  Log  Cmd,  McClellan  AFB,  ATTN:  ALC/DPCRB 

1 Air  Force  Academy,  CO,  ATTN:  Dept  of  Bel  Sen 
6 NavPers  Si  Dev  Ctr.  San  Diego 

2 Navy  Med  Nauroptvchiatrlc  Rich  Unit,  San  Diego 
1 Nav  Electronic  Lab,  San  Diego,  ATTN:  Ras  Lab 

1 Nav  TrngCen,  San  Diego,  ATTN:  Code  9000-Llb 
1 NavPostGraSch,  Monterey,  ATTN:  Coda  66Aa 
1 NavPostGraSch,  Monterey,  ATTN:  Coda  2124 
1 NavTrngEquipCtr.  Orlando,  ATTN:  Tech  Lib 
1 US  Dept  of  Labor.  DC.  ATTN:  Manpower  Admin 
1 US  Dept  of  Justice,  DC,  ATTN:  Drug  Enforce  Admin 
1 Net  Bur  of  Standards,  DC.  ATTN:  Computer  Info  Section 
1 Net  Clearing  House  for  MH-Info,  Rockville 
1 Denver  Federal  Ctr,  Lakewood.  ATTN:  BLM 
12  Defense  Documentation  Center 
4 Dir  Psych,  Army  Hq,  Russell  Ofet,  Canberra 
1 Scientific  Advsr.  Mil  Bd,  Army  Hq,  Russell  Ofca.  Canberra 
1 Mil  end  Air  Attache,  Austrian  Embassy 

1 Centre  da  Recherche  Das  Facteurt,  Humafne  dele  Defence 
Net  ion  ale,  Brussels 

2 Canadian  Joint  Staff  Washington 

1 C/Air  Staff,  Royal  Canadian  AF,  ATTN:  Pen  Sad  Anal  Br 

3 Chief.  Canadian  Def  Rsch  Staff.  ATTN:  C/CR0S<W) 

4 British  Def  Staff,  British  Embassy.  Washington 


1 Def  b Civil  Inat  of  Envlro  Medicine,  Canada 
1 AIR  CRESS,  Kensington,  ATTN:  Info  Sys  Br 
1 Mllltaerpsyhologlsk  Tlensste,  Copehagen 
1 Military  Attache.  French  Embassy.  ATTN:  Doc  Sec 
1 Medecin  Chef.  C.E.R.P  .A. -Arsenal,  Toulon /Naval  Franca 
1 Prln  Scientific  Off,  Appt  Hum  Enpr  Rich  Oiv,  Ministry 
of  Def  anas,  New  Delhi 

1 Pen  Rsch  Ofc  Library,  AKA,  Israel  Defense  Forces 
1 Minister  it  van  Defsntie,  DOOP/KL  Afd  Sociaal 
Psychotcglache  Zaken,  The  Hague,  Netherlands 
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